What is Machine Learning?

Why Machine Learning?

Machine Learning is transforming industries and daily life. Some key applications include:

AI, ML, and DL

Many people are confused what these terms actually mean.
And what does all this have to do with statistics?

Artificial Intelligence

Machine Learning

Deep Learning

ML vs Statistics

Type of Machine Learning

• Supervised Learning: learn a model from labeled data (ground truth)
Given a new input X, predict the right output y
Given examples of stars and galaxies, identify new objects in the sky
• Unsupervised Learning: explore the structure of the data (X) to extract meaningful information
Given inputs X, find which ones are special, similar, anomalous, …
• Semi-Supervised Learning: learn a model from (few) labeled and (many) unlabeled examples
Unlabeled examples add information about which new examples are likely to occur
• Reinforcement Learning: develop an agent that improves its performance based on interactions with the environment

Supervised Machine Learning

• Learn a model from labeled training data, then make predictions
• Supervised: we know the correct/desired outcome (label)
• Subtypes: classification (predict a class) and regression (predict a numeric value)
• Most supervised algorithms that we will see can do both

Type of Supervised Machine Learning

Supervised learning can be applied to two main types of problems:
• Classification: Where the output is a categorical variable (e.g., spam vs. non-spam emails, yes vs. no).
• Regression: Where the output is a continuous variable (e.g., predicting house prices, stock).

Regression

Regression is a type of supervised machine learning where algorithms learn from the data to predict continuous values such as sales, salary, weight, or temperature. For example:A dataset containing features of the house such as lot size, number of bedrooms, number of baths, neighborhood, etc. and the price of the house, a Regression algorithm can be trained to learn the relationship between the features and the price of the house.

Regression Algorithms

Linear Regression

Linear Regression is a supervised learning algorithm that is used to model the relationship between a dependent variable and an independent variable. The algorithm finds the best fit straight line relationship (linear equation) between the two variables. This statistical method is then used to predict the outcome of future events and is quite useful for predictive analysis.

Linear Regression:Example1

Linear Regression:Example1

Training Set

Linear Regression:Example1

Model Representation & Cost Function

Model Representation:

Cost Function(Hypothesis):

• Hypothesis: \(h(x) = ax+b\), where \(a\) and \(b\) are called parameters

Linear Regression:Example1

Cost Function(Cont.)

• Hypothesis: \(h(x) = ax+b\)

Linear Regression:Example1

Cost Function(Cont.)

Goal: Choose \(a\) and \(b\) so that : \(h(x) = ax+b\) is close to \(y\) for the training example \((x,y)\)

For each \((x^{(i)}, y^{(i)})\): Minimize \(|h(x^{(i)}) - y^{(i)}|\)

\[\Rightarrow \text{Minimize } \frac{1}{m} \sum_{i=1}^m |h(x^{(i)}) - y^{(i)}|\]

Squared Error Function:

\[J(a, b) = \frac{1}{m} \sum_{i=1}^m \left(h(x^{(i)}) - y^{(i)}\right)^2\] \[\Rightarrow \min_{a, b} J(a, b)\]

Linear Regression:Example1

Gradient Descent

Consider, we have cost function \(J(a,b)\), and our goal is to minimize \(\min_{a, b} J(a, b)\)

Linear Regression:Exercise 1

Land Price Prediction

Given a dataset of land price as illustrated in the table below, find a linear regression model which fits the data. Train the model using the gradient descent algorithm(implement from scratch).

Linear Regression:Example2

LR with Multiple Variables

Linear Regression:Example2

LR with Multiple Variables

Notation

- Hypothesis: \((h(x) = a x_1 + b x_2 + c)\)
or \((h(x) = \theta_0 + \theta_1 x_1 + \theta_2 x_2)\)
or \((h(x) = \sum_{j=0}^n \theta_j x_j\)\), where \((x_0 = 1) and (n = 2)\)
- Cost function: \((J(\theta_0, \theta_1, \ldots, \theta_n) = J(\theta) = \frac{1}{m} \sum_{i=1}^m (h(x^{(i)}) - y^{(i)})^2)\)

Linear Regression:Example2

LR with Multiple Variables:Feature Scaling

Make sure all features are on a similar scale

Linear Regression

Evaluation Approach

Train/Test Split (Data Generalization)

- The dataset is divided into two parts: training set and test set
- Training set: used to learn patterns and fit the model
- Test set: used to evaluate how well the model generalizes to unseen data
- Prevents overfitting and over-optimistic accuracy
- Analogy: Training = practice questions, Test = real exam

Linear Regression:Example2

Hyperparameter

Learning Rate

Learning rate is a floating point number you set that influences how quickly the model converges. If the learning rate is too low, the model can take a long time to converge. However, if the learning rate is too high, the model never converges, but instead bounces around the weights and bias that minimize the loss. The goal is to pick a learning rate that’s not too high nor too low so that the model converges quickly. The ideal learning rate helps the model to converge within a reasonable number of iterations.

Linear Regression:Example2

Hyperparameter

Batch Size

Batch size is a hyperparameter that refers to the number of examples the model processes before updating its weights and bias. You might think that the model should calculate the loss for every example in the dataset before updating the weights and bias. However, when a dataset contains hundreds of thousands or even millions of examples, using the full batch isn’t practical.

Two common techniques to get the right gradient on average without needing to look at every example in the dataset before updating the weights and bias are stochastic gradient descent and mini-batch stochastic gradient descent:

Linear Regression:Example2

Hyperparameter

Epochs

During training, an epoch means that the model has processed every example in the training set once. For example, given a training set with 1,000 examples and a mini-batch size of 100 examples, it will take the model 10 iterations to complete one epoch. Training typically requires many epochs. That is, the system needs to process every example in the training set multiple times. The number of epochs is a hyperparameter you set before the model begins training. In many cases, you’ll need to experiment with how many epochs it takes for the model to converge. In general, more epochs produces a better model, but also takes more time to train.

Linear Regression:Example2

Evaluation Approach

Error-Based Metrics (Predictive Accuracy)

Mean Absolute Error (MAE)

- Measures the average absolute difference between predicted and actual values
- Uses the same unit as the target variable
- Less sensitive to outliers than MSE

\[\text{MAE} = \frac{1}{n} \sum_{i=1}^{n} \left| y_i - \hat{y}_i \right|\]

Mean Squared Error (MSE)

- Squares prediction errors
- Large errors are penalized heavily
- Sensitive to outliers
- MSE is commonly used as a loss function during training because it is differentiable.

\[\text{MSE} = \frac{1}{n} \sum_{i=1}^{n} \left( y_i - \hat{y}_i \right)^2\]

Root Mean Squared Error (RMSE)

- Represents the typical size of prediction error
- Same unit as the target variable
- Sensitive to outliers
\[\text{RMSE} = \sqrt{\text{MSE}}\]

Linear Regression: Example 2

Evaluation Approach

Goodness-of-Fit Metrics (Model Explanatory Power)

R-Squared (\(R^2\))

- Measures the proportion of variance in the target variable explained by the model
- Compares the model against a baseline that predicts the mean
- Values range from 0 to 1 (higher is better)
- Does NOT measure prediction error

\[ R^2 = 1 - \frac{ \sum_{i=1}^{n} \left( y_i - \hat{y}_i \right)^2 }{ \sum_{i=1}^{n} \left( y_i - \bar{y} \right)^2 } \]

Adjusted R-Squared

- Adjusted version of \(R^2\) that accounts for the number of predictors
- Penalizes adding irrelevant features
- Essential for Multiple Linear Regression
- Increases only when a new feature improves the model

\[ \text{Adjusted } R^2 = 1 - \left( \frac{(1 - R^2)(n - 1)}{n - p - 1} \right) \]

Where: \(n\) = number of samples, \(p\) = number of predictors

Linear Regression: Exercise2

Linear Regression:Exercise2

Multiple Linear Regression

Given a dataset of land price:

1. Build a linear regression which predicts the land price using both the land_area and the distance_to_city feature. (See the dataset in ’land_price_ _1.csv’)
2. Using only the distance feature, build a model with hypothesis \(h(x) = \theta_{0}+\theta_{1}x+\theta_{3}\sqrt{x}\) to predict the land price. (See the dataset in ‘land_price_2.csv’)

Classification

- Predicts a class label (category), which is discrete and unordered
- Can be binary (e.g., spam / not spam) or multi-class (e.g., letter recognition)
- Many classifiers can return a confidence score per class
- Model predictions create a decision boundary separating the classes

Classification Algorithms

Logistics Regression

Logistics Regression

Hypothesis Representation

Logistics Regression

Cost Function

Logistics Regression

Cost Function(cont)

Logistics Regression

Exercise 1

Logistics Regression

Multiclass Classification

Exercise2

Questions?

Instinct Institute